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The junctional iterative approach given by Zador for calculating the 
average bit error probability in a regenerative repeater with quantized 
feedback is extended to the vector case. For a channel with a rational frac- 
tion transfer function, the vector extension permits us at least formally 
to deal with the following practical conditions'. 

(i) The pulse transmission plan is described by an m-ary alphabet 
with independent digits. 

(ii) Perfect and imperfect low-frequency tail cancellation cases are con- 
sidered. 

(Hi) High-frequency signal shaping and its interaction with the pre- 
dominantly low-frequency tail are taken into account. 

Expressions for error probability on the kth digit are derived in terms of 
the kth vector iterate of a known function. The restriction to independent 
noise samples is also removed. The resulting expression for kth bit error 
probability is then derived from an operational iteration procedure which 
acts on the k + 1 dimensional joint distribution of the noise samples. 

I. INTRODUCTION 

In the design of digital communication links, various reasons exist 
for the removal of low-frequency components during or prior to trans- 
mission of a pulse train. In the case of vestigial sideband (VSB) 
modulation 1 of data over voice-frequency channels, the dc and low- 
frequency signal components are removed at the transmitter before 
modulation and carrier reinsertion. This is required to insure satis- 
factory carrier recovery at the receiver for relatively low transmitted 
carrier power. In the T-l Carrier System, 2 the loss of low-frequency 
information results from transformer coupling of an unbalanced 

1S31 
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repeater to the balanced line. In either event, the effect of low-fre- 
quency suppression is to cause the positive impulse response of the 
overall equalized medium to exhibit an undershoot which gives inter- 
symbol interference. 

One means of reducing the effect of low-frequency suppression in a 
regenerative repeater is to feed back a signal in an attempt to cancel 
the long transient tail. This method of compensation has been called 
quantized feedback and its use dates back to the 1920's (as noted by 
Bennett 3 ). We assume that the reader is familiar with Bennett's ex- 
cellent expository paper. Until recently, analysis of the effects of 
quantized feedback on average bit error probability in a noisy en- 
vironment has received essentially no attention. The first to examine 
this problem were Anderson, Gerrish, and Salz* who considered the 
polar binary case, neglecting signal shaping and assuming perfect 
matching of the feedback cancellation signal to the input signal tail. 
They have obtained results, with the aid of the computer, that have 
provided insight into the problem. In addition, they have exposed com- 
putational difficulties involved in grinding out numerical results for 
any given set of system parameters. 

A more analytical approach to the basic problem is found in Zador 5 
who used the theory of generalized random jump processes 6 to obtain 
an iterative procedure for computing error probability. Unfortunately, 
the class of physical systems that can be handled by Zador's approach 
as originally stated is quite restrictive in the following sense (see 
Fig.l): 

(i) The transmitted message sequence is composed of independent 
binary digits. 

(ii) The low-frequency behavior of the channel as represented by 
G(s) is dominated by a single pole. 

[Hi) G(s) and H(s) are exact complements of each other so that 
perfect feedback tail cancellation is achieved. 

(iv) The time dispersion of the transmitted pulses caused by the 
medium, C(s), with or without equalization E(s) is strictly limited to 
two pulse intervals. 

(v) The noise samples at the input to the threshold detectors are 
assumed independent. 

It is our intention here to remove some of the above restrictions. In 
particular, we extend Zador's approach along the following lines: 

(i) By allowing a multilevel threshold device as a regenerator, the 
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Fig. 1 — Block diagram of reconstructive repeater with quantized feedback. 



allowable pulse transmission plan is extended to include wi-ary alpha- 
bets with independent digits. (The ternary case is treated in detail.) 

(ii) The high- and low-frequency behaviors of the channel may be 
individually characterized by rational functions. The implication of 
this is twofold. First, the predominantly low-frequency tail is now 
described by several exponentials. Secondly, the impulse response of 
the overall equalized medium C(s) E(s) is not restricted to be time- 
limited. 

(Hi) The restriction to perfect tail cancellation is removed to allow 
for imperfections in the forward and/or feedback paths. 

(iv) The more realistic case of correlated noise samples is examined. 

Extensions (i), (ii), and (in) are possible only through a vector ap- 
proach based on Zador's original iteration scheme. The assumption of 
a nonflat noise spectrum as in (iv) leads to an operational iteration 
procedure for calculating bit error probability. It is to be emphasized 
that the question of computational procedures, which even in the sim- 
ple binary case was a formidable task, grows considerably in com- 
plexity with the degree of generality assumed. 

The generalizations listed above will be treated one at a time so as 
to demonstrate individually the necessary changes in Zador's original 
formulation. A review of his model is given in Section II. 

Section III assumes an unrestricted ternary message sequence to- 
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gether with the remaining restrictions as imposed by Zador. An ap- 
plication of the results is given for a particular high-frequency behavior 
of the system. The response of the high-frequency portion of the chan- 
nel, C(s) E(s), to a transmitted rectangular pulse is assumed triangular 
in shape and time-limited to two pulse intervals. 

Section IV derives the general expression for error probability when 
the overall channel, Y(s) = C(s) E(s) G(s) is assumed to be charac- 
terized by a rational function. The feedback network, H{s), is de- 
signed to cancel only the low-frequency poles, i.e., those of G(s). The 
special case of a binary input format is treated in detail. 

Section V modifies the results of Section III by including the case 
of imperfect match of the G(s) and H(s) characteristics. 

Section VI begins with Zador 's original assumptions on the signal- 
ing format, channel, and feedback network characteristics, but re- 
moves the restriction of independent noise samples. An expression for 
fcth bit error probability is derived from an operational iteration pro- 
cedure which acts on the k + 1 dimensional joint distribution of the 
noise samples. The analogy between this scheme and the functional 
iteration proposed by Zador for the uncorrelated noise case is demon- 
strated. 

II. REVIEW OF ZADOR'S MODEL 

We begin with a brief review of Zador's mathematical assumptions 
and emphasize their physical significance. Consider once again the 
repeater-to-repeater transmission link illustrated in Fig. 1. The out- 
put of the nth repeater at time rT is a binary rectangular pulse* 
d r p(t—rT) where 

p(t) =Po \t\*U 
= 0, | / | > /o 

d T = ±1, and 1/T is the pulse rate of the system. Zador does not ex- 
plicitly describe the high-frequency behavior of the system. The class 
of channels that satisfies his underlying assumptions is discussed below. 
Let the response of C(s)E(s) to the pulse p(t), denoted by z(t), be time 
limited to 2T, and zero at its end points. It is understood that in practice 
these conditions are usually met only approximately. Then, by passing 
z(t) through a single pole high-pass filter, G(s), the part of the resulting 



* Zador assumes ±1 impulses as repeater output. As we shall see, in the sam- 
pled systems we consider, this modification has no effect on the ensuing analysis. 
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response, g(t), for t ^ 2T is dominated by a single exponential. If s(t) 
is sampled at time t = T and held until t = T + t , then by employing 
an ideal sheer element as a threshold detector a unit rectangular output 
pulse, b(t) is regenerated. Furthermore, by passing this pulse through 
II (s), the response tail of s(t) for t ^ 2T may be exactly cancelled in 
the absence of noise and circuit imperfections. These observations are 
illustrated in Fig. 2 for a triangular pulse shape z(t). The response of 
II (s) to the regenerator output pulse is denoted by h(t). Turning now 
to a sample notation, let g k , h k , and b k represent the values of g(t), h(t), 
and b(t), respectively, at time (k + 1)T, k = 0, 1, 2. Then, from Fig. 
2, it is obvious that the following conditions must hold, in general, in- 
dependent of the waveshape of z(t) within the 2T interval: 

(0 g > 0, h = 

(ii) h, + g, ■ = i = 1,2, •■- . 

(in) g, = rQi-i i ^ 2 




Fig. 2 — System pulse responses. 
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where r is related to the single pole, a, of G(s) by r = e~ aT . Con- 
dition (in) is clear upon noting that the response of a single pole high- 
pass filter to a time limited signal of width 2 T has a single exponential 
response for values of t ^ 2T. Statements (t) to (its) as above are 
identical with Zador's restrictions on the system as reported in Ref. 
5.* The shape of z(t) is solely used in determining the two dependent 
quantities g and g x . For a triangular z(t) waveshape of unity height 
(Fig. 2) and G(s) = s/s + a, 

9o = g; U " e~° T ] 
9i = -3; [1 " ■" 

III. TERNARY PULSE TRANSMISSION 

When considering a ternary system, the only essential modification 
of the model suggested by Zador is an ideal slicer with positive and 
negative pulse detection thresholds set at +a and — Oj, respectively. 

Letting s k denote the total reshaped input at the k + 1th timing 
instant, and c k the feedback voltage at the same instant in time as 
before, the slicing operation is described by 

b k = 1 if s fc + n k + c k ^ a 

= if —di < s k + n k + c k < a n 
= — 1 if s k + n k + c k ^ —a, , 



where 



- Eft-** k - 0,1 



c = 



= S**-,6, fc = 0, 1, 



and n fc is a sample from a stationary noise process n(t) having a fixed 
but arbitrary distribution function N(x), and independent samples. 
The process n(t) is actually the result of passing the additive white 
noise process in the system, ${t), through E{s). We assume, however, 
that the correlation between noise samples introduced by the above 
is small and can be ignored as a first approximation. When this as- 

- ; Note that Zador also requires fife < for » i£ 1. This restriction is not neces- 
sary although it is often true. 
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sumption is invalid, the method discussed in Section VI must be used. 
It is of prime interest to examine the conditions under which the 
system will operate error-free in the absence of noise. For < a„ , 
fli ^ Go , 

s + c„ = g d n 

thus if, 

a = 1 , s„ + c = Qn , &o = 1 

= 0, s + Co = 0, 6 = 

= — 1 , So + Cq = — (ft, , 6 = — 1 , 

or & = d . 

Continuing, in this way k = 1, 2, • ■ • , k — 1, 

' -i 
s* + c k = g„d k + J2 (g k -i + fc fc -,-)d,- 

1=0 

= g d k . 

Thus, if b m = d m for m = 0, 1, ■ • • , k — 1, then b A - = <2 fc and the sys- 
tem operates error-free in the absence of noise. 
For the more general case when noise is present, 

s* + c k = 2 0*-j(rf< - &*) + g<id k = x t + porf* , 

i-0 

where a^ represents the cumulative effect of any and all errors prior 
to time k. 

Letting p and q denote the a priori probabilities of a plus one and 
minus one, respectively, the probability of error on the fcth digit p(k) 
can be written as 

p(k) = p Prob [n k + x k ^ a — g ] + q Prob [n k + x k > -a, + g„} 

+ (1 — p — q) Prob {n k + x k ^ a ; n k + x k ^ —a, } . 

The independence of n t and .r t allows p(fc) to be expressed in terms of 
the noise distribution function N(x) and the distribution function of 
•T/.-, F k (x) as follows: 



p(k) =p f N(a - g a - x) dF k (x) 

J-ao 
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+ q r [1 - N(g -a,- x)] dF k (x) 

J-oo 

+ (1 - p - q) r [N(- ai - x) + 1 - N(a - x)] dF k (x). 

J — oo 

For the case of a zero mean symmetrical noise distribution and equal 
& priori probabilities for all input symbols (i.e., p = q = (1 — p — q) it 
is easy to show that the optimum threshold settings are ±g /2 with 

p(k) = (1 - p) P [1 - N(g /2 - x) + N(-g /2 - x)) dF k (x). 

J-OO 

It now remains to show that the sequence of random variables 
Xo , %i , ' ' * are representative of a random jump process studied in 
Ref. 6 and thus p(k) can be expressed as the fcth iterate of a known 
function evaluated at Xo with a finite limit as k -» oo . 

Consider, 

Jb 

x k+ i = J2 9k + i-i(di - &,•) 

t" = 

k-l 

= g,(d k - b k ) + 2>0*-<(<*< - W 

»=0 

x k+ i = gi(d k — b k ) + rx k . 

There are five possible transition states each of which takes place 
with probability depending on the value of x k . 

If d h - 1, &* = -1, then x t+ , = rz* + 2p, with probability pxfo). 
If d* = 1, b k = 

or , then .r t+I = rx k + £, with probability p 2 (x k ). 

d k = 0, b* 1 

If d* = b k , then .r t+1 = rx k with probability p 3 (^)- 

If d k = -1,6* - 

or , then .t a . + 1 = rx k — g x with probability p t (x k ). 

d k = 0, 6, = 1 
If d* = — 1, b k = 1, then z t+1 = rz t - 2^, with probability p 5 (x k ). 



REGENERATIVE REPEATERS 1839 

The transition probabilities p n (x k ), n = 1, 2, • • •, 5 are denned by 
p,(.i\) = pN(-a t - g - x k ) 
?>>0a) = p[N(a - g ~ x k ) - AT(-a, - g - x h )] 

+ (1 -V ~ 7)[A r ("«. - **)] 
p a (x k ) = 1 - p^x k ) - 7>,(.r A ) - p 4 (x k ) - p 5 (x k ) 
Pi(x k ) = q[N(a + g - x k ) - N(-a t + g - x k )] 

+ (1 - p - g)[l - N(a - x k )] 

Ps(x k ) = g[l - iV(a + 0o - **)]. 

Note, 

p(*D = /"" b.(-r) + p 2 (z) + p 4 (x) + p 6 (s)] <^(*)- 
Defining ^[/(.r)] = Pi(x)f(rx + 2g,) + p 2 (x)f(rx + g x ) + p 3 {x)1(rx) 

+ P4(.-c)/(^ - £.) + P5(*)/(Vz - 20,) 
and denoting the /cth iterate of U l \j(x)] by E/*[/(x)], 

p(Jc) = U k \p,{x) + p 2 (s) + PM + p,(.r)] |.-.._ • 
If A(x) is the limiting distribution of F k (x), then 

Vim p(k) = /" [ 7 ;,(:r) + p*(x) + p 4 (*) + p 5 (*)] dA(x) 

k— 00 •'-00 

= lim C/*[p,(.r) + ?>-•(•'•) + P*(x) + Psfr)] |,-,.- • 

t— 00 

A few remarks are now presented to indicate the obvious extension to 
the ??i-level (m-ary) pulse transmission scheme. A random jump proc- 
ess with 2m — 1 transition states will result requiring an iteration func- 
tion U x [f(x)] having 2m — 1 terms. It should be indicated that com- 
putationally the amount of computer storage or operations required to 
evaluate p(k) is of the order (2m — l) k . 

IV. RATIONAL FUNCTION APPROXIMATIONS OF THE CHANNEL AND 
FEEDBACK NETWORKS 

As the subtitle indicates, we are interested here in studying the 
repeater error performance under the assumption of a rational func- 
tion approximation to the channel and feedback networks. This gen- 
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eralizes the assumptions of Section III in that (i) the tail of the pulse 
response, g(t), is no longer described by a single exponential, and {ii) 
the high-frequency behavior of the channel allows its time response to 
exceed two pulse intervals. To isolate these effects, however, perfect 
feedback tail cancellation is still assumed and we return to a binary 
message format. 

It is convenient to represent the output rectangular pulses of the 
nth repeater as the impulse response of a filter F(s) = (l/s)[l — e~"°] 
where t n is the pulse width. Including this filter in the forward path 
of Fig. 1, the overall channel link between repeaters, T(s) = F(s)C(s) 
E(s)G(s), is assumed to be characterized by a rational function as 
follows : 

P(s) 



T(s) = Go -, X 

ii (« + «<) ii (« + a) 

1=1 >-l 

with its associated impulse response 

g(0= E^— ' + EV". 
i=i t ■ i 

Note, the impulse response of T{s) is the same as the rectangular 
pulse response of Y(s) = C(s) E(s) G(s) and is thus denoted as be- 
fore by g(t). All poles are assumed to be simple, but in general may be 
complex. The terminology used henceforth will refer to the set {«*}, 
i = 1, 2, • • • , M as low-frequency poles and the set {/?,}, t = 1, 2, 
• • • , N as high-frequency poles. The inference here is that the ft's are 
predominantly responsible for signal shaping and the at's determine the 
low- frequency cutoff of the channel. 

A low-pass quantized feedback path II (s) is proposed which in the 
absence of noise would provide perfect low-frequency tail cancellation 
at all sampling instants beyond the input pulse peak (the effect of im- 
perfect low-frequency compensation will be discussed in Section V).* 
Thus, if 

H(s) = H — e , 

n c« + «<) 

i=i 

where t represents the physical delay in the feedback path beyond the 
occurrence of the input pulse peak at t = t mnx , then, the response to a 



* It is to be emphasized at this point that all of the following is easily gener- 
alized in terms of MacColl's conception of quantized feedback 7 wherein restora- 
tion of both low- and high-frequency signal components is attempted. 
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positive regenerator output pulse at t = t max would be 

h(t) = £ Z)*-'"- «-'.—> = £^-» for / ^ /_ + ro + * • 

.-1 ,• = i 

Ideally, for perfect low-frequency tail compensation, we desire 
E-E,e-°" + E^ge-' 1 = 

i - 1 i=l 

at all instants t umx + nT, n = 1, 2, • • -, where 7 1 is the uniform sam- 
pling period. 

Letting ft* and g k represent the values of the pulse responses h(t) 
and g(t), respectively, at the feth sample point the above statements 
may be expressed in brief as follows: 

(i) h„ = g > 

(ii) hi + Qi = £««., i * 

F.-l 

[n = 1, 2, ••• ,N 



0i.ll = Z,A-1. n- 



[i > 2 



e .„ = 



2„ = e 



-3,T 



(m) 



a. = £)*«.. 



/*..„ = ''„/!,_!,„. 



f/i = 1,2, ••• ,M 
I* > 2 



The term e< ,„ represents the residual intersymbol interference at the 
ith timing instant due to the nth high-frequency pole. To simplify 
what is to follow and at the same time allow a better comparison with 
the previous work of Zador, we introduce the following vector nota- 
tion : 

r 



R = 



r, 

/•.••• 



z = 



•>'.vj 



0' 







■z.v_ 
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// 



ho.y fti.i A3.1 /'3.1 ••' ^*.l 

/in. 2 fei.3 ^2.2 h 3 , 2 ••■ h k ,a 

Jlo.M h\,M h 2 ,M ha,M "' hk,M- 



G = 



0n.i 


0..1 


02.. 


03.. 


0fc,1 


00.2 


0..2 


g-j.2 


03.2 


0*".2 



-0O..W + .V 01.J/ + .V 02..V + .V 03..W + .V ••■ 0A.A/+.V- 

e lA e 2 ,x e 3i i ••• e*.i 
e 1-2 e 2 , 2 e 3 . 2 •■■ Cj...-» 



5 = 



.0 



e 3 .,v 



e k ,N- 



Using H as an example, the ith row written as a column vector is de- 
noted by h' and the ith column by the vector h ; . Also any vector written 
not in bold face is by definition the scalar representing the sum of its 
elements (e.g., ti = £l-i h Pli ). Finally, we denote the column vector 
obtained by summing all rows of H (i.e., whose ith component is h { ) 
by h. All of the above statements are equally applied to the matrices 
R, Z, G, and E. 

In terms of the above, (i), (ii), and (Hi) may now be rewritten as: 



(0 

(ii) 

(Hi) 



h = 0, 
h + g = e 
d = Zei-i, 



0o > 



i > 9 



hi = Bhj.i, i ^ 2. 



Some further interpretation of the above statements in terms of 
the actual system operation might prove helpful at this point, (i) 
indicates a positive input pulse peak (g > 0) and a delay in the feed- 
back path (h = 0). Statements (ii) indicate that perfect feedback tail 
cancellation is achieved at the sample points starting with the second 
except for the effects of the high-frequency poles (/3, , ft, , • • • , p N ). 
In contrast to the previous sections, we do not assume that the high- 
frequency components of the response have died out before the oc- 
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currence of the next input pulse peak. Feedback cancellation of only 
the channel low-frequency poles is described by statements (Hi). 

For the binary message case, the threshold detector box of Fig. 1 
reduces to a simple ideal sheer element operating between +1 and 
— 1 levels. The input sequence {d,| is a random train of +1 and —1 
impulses represented by the vector d with elements d { . 

The total reshaped input at the k + 1th timing instant, s k , and the 
feedback voltage at the same instant, c k , are described by, 

S * = (d*g). k = 0th2t ... } 

c k = (b * h) k 

where the A;th element of b, b k = sgn {s k + c k + n k } is the fcth regenerator 
output digit. The notation (a * b) fc represents the convolution of two 
k + 1 dimensional vectors a and b (i.e., ^< =0 a,6 A ._,). 

Considering first operation in the absence of noise, we see by in- 
spection b = d . (This tacitly assumes that no intersymbol interference 
due to precursors is present.) Proceeding as in Zador, 5 if b m = d m for 
m = 0, 1, • ■ • , k — 1, then 

s k + c k = (d * e) k = g a d k + (d * e) t _, . 
From this, one concludes that if 

g > J2 I e,- I, 

. = 1 

then b k = d k and the eye is open. The system will therefore operate 
error-free in the absence of noise for any length input sequence if 

°C 

9u > L k I- 

If all iV high-frequency poles (ft , /3 2 , • • • , f} N ) have positive residues, 
the above criterion reduces to 



->£r^7 



The above implies that the eye is open if the total high-frequency 
contribution at all sample points beyond the first is smaller than the 
pulse peak. 

More specifically, the values of e Un and z n may be related to the 
allowable amount of degradation of the eye. That is, for any eye which 
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is X percent closed. 



100 






On 



Turning now to the more realistic situation in the presence of noise 
s* + c k = g Q d k + (d * g),_, + (b * h)^ . 

Consider subdividing the vector d into two parts d' and d" in such 
a way as to separate the input digits into two classes corresponding 
to bi = di and &,• 5^ d t respectively. That is, 

d' t = d t ; dY = [%) 6, = d t ) 

= = d t \i; h 5* di}. 

(Obviously d = d' + d".) 
Then, using (ii), 

u, = (d*^Vi ; v * = -2(d*H r ) A ._ 1 

s k + c k = g d k + u k + v k 

= g„d k + x k , 

where (d * G) k - X is a vector whose ith component is the convolution of d 
with the ith column of G. Again omission of the bold face notation indicates 
summation over all the components and T is the transpose operator. 

The first term in x k denoted by u k represents intersymbol inter- 
ference due to residual high-frequency tail components irrespective of 
previous decisions. The second term v k again represents the cumulative 
effect of any and all errors prior to time k. 

The expression for error probability on the fcth digit is identical 
to that given by Zador, namely, 

p(k) = V f X N(-g - *) dF k (x) + q [ [1 - N(g - x)] dF k (x). 

The only difference being the nature of the distribution function F k (x). 
The recursive properties of the intersymbol interference x k are now 
examined. 

x M = (d * E T ) k - 2(d" * H T ) k . 
If b k t± d k , then 

.r, +l = («, - 2h 1 )d k + z r (d*A-. ~ 2r r (d" * H T ) k . 1 
= (e, - 2h 1 )d k + z r u k + r r v k . 
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If b k = d k , then 

x M = e,d* + z r (d *E T ) k . i - 2r r (d" * 7/ r ),_, 

= c A d k + z 7 u k + r r v k . 

Letting a = — 2h, , the intersymbol interference sequence .r , .r, , 
.r 2 , • • • may be expressed as a random jump process, 5,0 with the following 
transition states: 
If dk — 1 7* b k , then with probability Pi(x k ) 

x k + 1 = z r u k + e, + r r v k + a. 
If di, — 1 — b k , then with probability p 2 (x k ) 

x k+1 = z T u k + e, + r r v k . 
If d k = 1 ^ b k , then with probability p 3 (x k ) 

Xt+i = z T u k - c, + r r v k - a. 

If d k = —1 = b k , then with probability Pi(x k ) 

x* +] = z^Uk - c, + r 7 v k , 
where 

?>.(•'•/) = pN(-g„ - .r,) 
P..(.r*) = p[l - A T (-f7u - .T t )] 
p 3 (xi) = q[I ~ -V(0<j - x k )] 

P&t) = 5[-V(^o - Xk)]. 

In the above, X(x) is the distribution function of the stationary noise 
process, and p and q are the a priori probabilities of a plus one and 
minus one, respectively. In terms of the above elementary probability 
density functions, the error probability on the fcth digit may be ex- 
pressed as: 



p(k) = P b,(x) +p 3 (x)]dF k (x). 

•'-co 



AVe propose a vector extension of Zador's procedure, namely; an M + N 
dimensional iteration scheme in which each of M + N variables is re- 
placed by a linear transformation on itself during each iteration. To 
elucidate the meaning of M + N dimensional iteration and at the same 
time recall some of our earlier vector notation, the first-order itera- 
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tion function C/ 1 / is written in summation notation as: 



IN m \ r N 

tf 7(e, v ) = pil Z '. + Z *-) •/ Z W» 

\, ( 1 m-1 / l_n = l 



+ 0+ Z fr»ft.+ oj 



+ Pa (Z 

\ .1-1 
+ Pa(Z 



« \ r ,v ^ j 

+ Z *»-.)■/ EW. + 0+ Efr-ft.) 

+ Z <P m )-l\t(^ n - O + Z (r^, n - Ol 

,„-l / Ln = l W-l J 



+ P* Z ft. + Z ft.)-/ Z M. " O + Z (r m ft) • 

\„-l m = l / Ln = l m = l -J 

It follows that the probability of error on the fcth digit is 

p(k) = L Tt 'b. +v-A k,-o 

where U k is the /cth M + N dimensional iterate of U 1 . The convergence 
of p{k) in the limit as k — » <x has not been examined for an M + N 
dimensional branching process. From Zador's work on one-dimensional 
branching processes we may conjecture that absolute system stabil- 
ity (i.e., all poles in left-half plane) implies convergence in the multi- 
dimensional case. 

Although the notation in the foregoing analysis appears formidable 
(quite an understatement) the procedure and its usage are straight- 
forward (at least analytically) for a particular example. At the ex- 
pense of being redundant, we once again point out that even in simple 
cases, numerical results are hard to come by. 

V. IMPERFECT LOW-FREQUENCY TAIL CANCELLATION 

It is relatively simple at this point to include the effect of imperfect 
low-frequency cancellation in the results of Section IV. As an example, 
such a phenomenon might be caused by a delay of amount t in the 
feedback path. Defining an L matrix by 



L = 



/,.. h.x h. 1 
/,, 2 L.2 l a% * 

_0 l\.M h,M h.M 



h.y 
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where 

l\,m = hi, m 1 — exp ( — jr } log, — ) m = 1,2, • • • , M, 

statement {ii) of Section IV may be modified as follows: 
(ii) h + g = e + 1 

li = RU-t i ^ 2 
e { = Zei-i . 
The effect of this on the recursion relationship for x k is as follows: 
If b k 7* d k , then 

x k+l = (e, + 7, - 2Ai)d t + z'un + r r v k + r r o) k . 
If b k = d k , then 

s*+i = (e, + kK + z r u k + r r v k + r r » k , 
where to k = (d * L r ) t -i . 

If d fc = 1 t^ & t , then with probability Pi(.r A .) 

a-* + , = z r u k + c, + r 7 v k + a + r r a> k + Z t . 

If d 4 = 1 = bh , then with probability p 2 (.r fc ) 

a- A+1 = z r u k + e, + r r v k + r r w k + /,. 

If d* = 1 ^ 6 t , then with probability p 3 (.r 4 ) 

&*+, = z 7 'u k - c, + r r v k - a + rV - t. 

If d 4 = — 1 = b k , then with probability Pi(x k ) 

Xt-n = z r u k - e, + r 7 v k + r r o k - Z, , 

where PiOejO, P2 (•?*)> P3(£*)> and p.i(.T fc ) are still defined as in Section IV. 
The /cth bit probability of error is now evaluated by a 2M + A r 
dimensional iteration scheme where the first-order iteration function 
U l j is written as 



u i m ^ Y ) - pit, <>n + it v.» + Z Y.) 



£ M. + O + 2 frrf- + O + 2 (r.T. + «i. 



,.,] 
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/ V M M 

+ P.(2>.+ !>- + St. 

\„ = 1 m = l m-l 

■/[em. + o + f; (r^ m ) + z fa* + /..») I 

Lb = 1 m-l m = l J 

I V V JU \ 

\ n = 1 m-l »i-l ' 

•i\ib M - O + £ fo*- - a «) + 2 Gvr- - O 

Ln=l m = l m-l -I 

/ If IU if \ 

\„ = i »i = i m-1 ' 

Jib m - o + s fr-io + £ fr.T. - ol- 

Ln-l m = l "1 = 1 -J 

It once again follows that the probability of error on the kth digit is 

p(k) = t/ A 'bi + P*\ |.,,. T -o , 

where U* is the fcth 2M + N dimensional iterate of U\ 

To reward the reader for his patience up to this point, we will at 
least demonstrate that the general expression for p(k) given above 
reduces to Zador's result for the single low-frequency pole, perfect 
cancellation case. The assumption of no high-frequency signal shaping 
and perfect cancellation imply that 0, d , and y, U are, respectively, 
zero. Furthermore, a single low-frequency pole results in r, , <p x , and 
a x being the only nonzero components of r, «£, and a, respectively. Under 
these conditions, 

p(k) = U'lpx +Ps] I,,-., 

where 

U l f = PiidKr&i + a,) + P>(<Pi)f(ri<Pi) 

+ P:MJ(r 1V1 — a,) + pMKr&i) 
which is identical to Zador's result upon combining p^in) and pi{<?\). 

VI. THE EFFECT OF NOISE CORRELATION 

In this part, the emphasis is placed upon removing the restriction of 
uncorrected noise while at the same time arranging the results in a 
form which allows easy comparison with the uncorrelated case. The 
approach to be followed is the reformulation of Zador's work into 
an operational iteration procedure which acts on the joint distribution 
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of the noise samples. The details are presented for the simple binary 
case with perfect feedback cancellation considered by Zador. With 
sufficient patience, extension to the more general situations covered in 
the foregoing sections can be accomplished, but that is not done here. 
To review, the operation of the simplified system may be described 
by the equation 

b k = sgn {n k + god k + x h ], (1) 

where 

fc-i 

X„ = 2 £ g k _4 t (2) 

i=0 
bi—di 

represents the intersymbol interference accumulated at time t k as a 
result of errors (d { ?* 6.) prior to that time. 
The system output b k is in error when 

n k + x k < —g and d k = 1 

n k + Xh > 9o and d k = - 1 . 

Since the noise samples are not assumed to be independent, the random 
variables n k and x k are not independent. Hence, the distribution of the 
effective noise n k + x k is not simply the convolution of the distributions 
of n k and x k . Instead, the expression for error probability p(k) = 
prob { b k t^ d k \ must be written as 

I m,(n k , x k ) dn k dx k 

-00 * — M 

+ q I / m 2 (n k , x k ) dn k dx k , (4) 

where m 2 (n k) .Ta) is the joint density function of n 1c and x k and p and q 
are the a prion probabilities of a +1 and — 1, respectively. 

A careful examination of the branching process described in Refs. 5 
and 6 for the uncorrected case shows that a similar process governs 
in the correlated noise case. Define the integral operators pi (x) , p 2 (x) , 
p 3 (x) by 

p t (x) = p J 

P,(x) = J -P J ~ 1 ] (5) 

Pafr) = q / 
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Note that the action of each of these three operators on a single di- 
mension Gaussian density function results in the three transition prob- 
abilities defined by (13) of Ref. 5. If f(x) is denned analogously (but 
operationally) as 

m = Pi(*) + p a (*) =v f ' ' + q T . (6) 

then the first-order iterative operator Uf(x) is expressed as in Zador, 
namely : 

TO = Pi(*)/(« - a) + p 2 (x)f(rx) + p 3 (x)j(rx + a) (7) 

with a = —2g l . We note that after separating f(x) into its two com- 
ponents parts, each term of (7) represents a double integration and 
thus (7) has meaning only when applied to a second-order density 
function. Proceeding as in Zador, the error probability on the fc+lth 
digit in a random input sequence is expressed as the /cth iterate of the 
operator Uf{x) acting on the fc+1 dimensional joint density function 
of the noise process v,.+i (yi, yo, ' ' ' , yic+i) evaluated at x = 0, i.e., 

p(k) = U k Kx)[v b ( 7l , y 2 , • • • , y, +1 )] |x-c* (8) 

The meaning of iteration for the operators defined here is the same 
as in Zador's functional case. As an example, we write out p(l) in 
detail: 

/-Bo r>-0<i + a 
J y,(yi , y 2 ) dji dy 2 

+ pq I / »«(ti . 72) dji dy 2 + p I I y s (yi , y 2 ) dy { dy 2 

• '-co Jo„+a J-<a J— to 

+ g I / v»(y t , y 2 ) dy, rfy« ~ P 2 j j »(ti > 7z) ^Ti ^72 

- pq / y 2 (yi , y 2 ) dyi <fy 2 - pq I / o«(7i . 7z) <*7i dy 2 

«— CO ■'to "S» "—00 

,»co noo /»°0 p — Vo — a 

- q 2 / «z(7j , 72) dyi ^72 + pq / / «a(7i . 72) drti dy 2 

"Bo "Bo 0° "—00 

+ q 2 f M r Vrfyt , y 2 ) dn dy 2 . (9) 

■'so J Oo-a 

* The convergence of the operational iteration procedure denned by (7) and 
(8) haa not yet been proven. Nonetheless, we proceed with our results. 
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The above expression for p(l) can be simplified for a symmetric den- 
sity function v 2 . It is further emphasized that the arguments of each 
term in the operator as defined by (7) determine the limits on the 
integrals in (9) (i.e., the region of integration) . 

VII. CONCLUSIONS 

The analysis presented in this paper might in a broad sense be 
described as vector and operational extensions of the work of Zador. 
In addition to simply considering a vector of low- frequency poles, 
however, the vector approach has enabled us to remove certain other 
restrictions from the basic regenerator problem such as lack of high- 
frequency signal shaping and perfect tail cancellation. Although, the 
question of convergence of the operational iteration scheme for cor- 
related noise samples remains as yet unanswered, the formulation itself, 
is of interest. Little has been suggested for solving the exact computa- 
tional problem. A future paper will discuss some useful approximations 
to cases of relatively low dimensionality. This will generalize results 
given in Ref . 8. 
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